[SPARK-27528][SQL] Use Parquet logical type TIMESTAMP_MICROS by default #24425
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #104775 has finished for PR 24425 at commit
|
|
Test build #104776 has finished for PR 24425 at commit
|
MaxGekk
changed the title
felixcheung
reviewed
Apr 21, 2019
Member
felixcheung
left a comment
felixcheung
left a comment
There was a problem hiding this comment.
could we document this? persistence data format changes are potentially fairly impactful
|
Test build #104782 has finished for PR 24425 at commit
|
HyukjinKwon
approved these changes
Apr 22, 2019
Member
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
Looks good given that INT96 is deprecated in Parquet too, and apparently migrating looks fine 777b797
|
Test build #104797 has finished for PR 24425 at commit
|
|
Test build #104800 has finished for PR 24425 at commit
|
Member
|
Merged to master. |
rshkv
added a commit
to palantir/spark
that referenced
this pull request
Mar 9, 2021
The upstream default is INT96 but INT96 is considered deprecated by Parquet [1] and we rely internally on the default being INT64 (TIMESTAMP_MICROS). INT64 reduces the size of Parquet files and avoids unnecessary conversions of microseconds to nanoseconds, see [2]. Apache went down the same route in [2] but then reverted to remain compatible with Hive and Presto in [3]. [1] https://issues.apache.org/jira/browse/PARQUET-323 [2] apache#24425 [3] apache#28450
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In the PR, I propose to use the
TIMESTAMP_MICROSlogical type for timestamps written to parquet files. The type matches semantically to Catalyst'sTimestampType, and stores microseconds since epoch in UTC time zone. This will allow to avoid conversions of microseconds to nanoseconds and to Julian calendar. Also this will reduce sizes of written parquet files.How was this patch tested?
By existing test suites.